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ABSTRACT 

This paper discusses intellectual competence and ways 
in which to define, measure and use this concept in evaluating 
college effectiveness. Intellectual competence is divided into two 
categories; academic mastery and intellectual resourcefulness, with 
evaluation methods suggested for each. (CK) 
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INTELLECTUAL COMPETENCE: DEFINITION AND MEASUREMENT 1 

Richard E. Peterson . 

Educational Testing Service 

In the past year I have had occasion to do some systematic thinking 
about college and university goals. In connection with work on an 
Institutional Goals Inventory to be used by colleges in identifying 
goals and setting priorities among them, Barry Morstain and I developed 
a working conceptualization of the domain of higher education goals, which 
breaks down into 13 "output" goals and nine "process" goals. The first 
two output "goal areas "“-central aims for most colleges and universities, 
one may assume--have to do with student academic and intellectual develop- 
ment, and these are the matters I want to focus on in this paper. More 
exactly, I want to discuss a concept --name it, define it, discuss how to 
measure it, and comment on its use in evaluating the effectiveness of 
colleges . 

Intellectual Competence: A Definition 

The name I've given the concept is Intellectual Competence. How shall 
it be defined, such that the definition will both make sense given the 
nature of the times, and lend itself to credible measurement? To begin, I 



1 Basfed on a talk given at 'the 12th American Meeting of the Institute •. 
of Management Sciences, Detroit, September 30, 1971* 

^Output goals: Academic Development, Intellectual Orientation, Individual 

Personal Development, Humanism/Altruism, Cultural /Esthetic Awareness, 
Traditional Religiousness, Vocational Preparation, Advanced Training, 

Research, Meeting Local Needs, Public Service, (Social Egalitarianism, Social 
Criticism/Activism. 

Process goals: Freedom, Democratic Governance, Community, Intellectual/ 

Esthetic Environment, Collegiate Environment, Innovation, ; Evaluation, and 
Planning, Accountability/Efficiency, External Relations . 
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would suggest that we can usefully think of intellectual competence as com- 
prised of two basic components, which roughly correspond to the first two 
goal areas in the conceptualization referred to above. 

The first component may be called Academic Mastery. While the relative 
emphases may vary from college to college, academic mastery would generally 
embrace, first, in-depth understanding in one field; second, a more modest 
acquaintance with facts and principles of the western intellectual heritage; 
and, third, basic literacy--the ability to read, write effectively, and do 
simple mathematics. Nothing particularly surprising so far. But how many 
colleges, one wonders, in fact award degrees on the basis of some level of 
demonstrated mastery, let alone evaluate their effectiveness in terms of 
the amount of gain in knowledge ability. 

The other component of general intellectual competence I would like to 
call Intellectual Resourcefulness. What it involves, fundamentally, is the 
capacity, in a social context, to solve problems. Its importance as an 
"outcome" of college, of course, owes to some of the likely social "givens" 
of the last half of the 20th century--social, technological, biological 
change; accelerated obsolescence of knowledge and vocational proficiencies; 
and so forth. 

Among the specific skills of the "intellectually resourceful" individual 
might be the following: 

(l) Skill in problem definition: ability to identify the forces that 

interact to define the problem, to systematically observe and empirically 
assess, to "transcend" the problem situation (minimize the intrusion of 
one ' s own values ) . 
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(2) Capacity/confidence to engage in new learning: ability to come to 

understand and be able to evaluate problem inputs from theretofore unfamiliar 
sources, skill in actively seeking out new kinds of information. 

(3) Creativity in proposing problem solutions: ingenuity, ability to 

comprehend all relevant variables , capacity to forecast consequences of 
alternative solutions. 

(4) Capacity for cooperative work: ability to function effectively as 

a member of an interdisciplinary task group: patience, tolerance, respect 

for divergent beliefs. 

Again, much of this is not new to people who have thought about the 
problem of problem solving. But I doubt that many colleges have seriously 
considered some standard of ability to solve problems, however this may be 
defined, as an institutional goal, much less as a specific exit criterion.^ 
Both comp one nts--knowledgeability and ability to put the knowledge (and 
new knowledge) to work in solving problems on the job and in everyday life-- 
seem essential as college goals. Assuredly, one cannot apply knowledge to 
the problems of life unless one has it to begin with. Just as assuredly, it 
is the person who has developed a resourcefulness in solving problems and 
continuing to learn who will be able to keep his sanity as the future comes 
crashing down on us . 

So much for conceptualizing. What about measurement? 



This leads, of course, to questions about the probable worth of more 
or less formal instruction in problem-solving techniques, per s e, whether 
or not such skills can be learned separate from the content of particular 
disciplines, and so forth. My own belief is that an interdisciplinary "course" 
(or sequence, offered during the undergraduate years in which increasingly 
difficult problems are considered) addressed generally to the problems of 
living and working in a complex society is both feasible and valuable. An 
alternative would be separate courses that emphasize information (very broadly 
defined) from, say, the social sciences, the natural sciences, and the 
humanities and arts. 

erJc 4 
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M easuring Intellectual Competency 

With regard to "academic mastery," there are a host of available tests 
for assessing subject matter achievement or mastery, in depth and/or in 
breadth. And there are published tests that can be used to assess basic 
literacy. 

Let me, however, suggest an alternative approach to the use of published 
tests in measuring specialized field proficiency, that is, for measuring 
knowledgeability in one's major field. It involves the use of what in the 
past were called "senior comps "--comprehensive exams given at the end of 
the senior year. The suggested procedure is for faculty, administrative 
and student representatives in a given discipline at the college, or from 
all the institutions in a multicampus system, to work together to jointly 
define the domain of knowledge, mastery of which would be requisite for the 
degree (e.g., a BS in chemistry, an AB in business administration).^ The 
resulting general specifications for a degree examination would be turned 
over to some other agency--say, a statewide university examiner- -which 
would construct the actual test. Professors .and students would return to 
their respective departments to develop the most effective set of learning 
experiences they can conceive for instilling mastery of the subject domain 
as they have defined it. (The profs would never see the test itself.) 

This whole process of degree-definition might '-be ;repeated every five years' 
or so. 



The difficulties involved in a departmental faculty reaching consenses 
about the objectives of instruction in the department can hardly be over- 
emphasized. At many universities, the impasse regarding (departmental) 
curriculum design is such that faculty simply refuse to talk about it. (The 
prospect of using some published test, e.g., the GRE, could be a stick to 
prod departments to reach agreements about their learning objectives.) 
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As for measures of breadth of knowledgeability, there are available what 
are called "area" tests in the social sciences, natural sciences, and humanities 
There are long, highly reliable area tests, and there are also short versions 
suitable for assessment of group performance. And while there are many avail- 
able standardized tests of ability to read, write, and compute, the college's 
English and Math departments should be able to devise these instruments. 

Technically, the task of preparing measures of different kinds of academic 
mastery is not all that difficult. The situation, however, is quite different 
with respect to the kinds of skills I outlined under the idea of "intellectual 
resourcefulness." In fact, the state of the measurement art is still such 

that there are no measures of adult "problem-solving ability" or "creativity" 

5 

that I could recommend. Here, it seems to me, is a challenge to the 
psychological measurement fraternity, ETS included. Given an appropriate 
marshalling of expertise and money, I'm reasonably confident the challenge 
could be met . 

Basically, I would think these tests would be in the nature of what are 

called situational" tests, or tasks, or games. For example, it should be 

6 

possible to devise one or more problem sequences in which a situation- -data, 
prose summaries of political and economic farces, etc., for example--is 
presented, perhaps by means of slides for group administration, with the 
students asked to write out an answer to the question: "What would you do?" 

: 5 

Some of the advanced college achievement tests (e.g., some of the 
GKEs) contain items that tap students' ability to apply facts and principles 
of the discipline to certain kinds of problems. 

6 

Different problem solution exercises could be framed in terms of 
several broad disciplines such as the social sciences, natural sciences, 
physical sciences, business, education, social ethics, and so forth. 
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This sequence would be repeated perhaps twice, with successively new data 
introduced that would call for modified or new solutions. The entire sequence 
might take 50 minutes . 

Another approach, developed at ETS and used quite widely in industrial 
settings, is known as the "in-Basket" test. Incoming "memoranda" present a 
problem; the subject records his decisions regarding the problem in out- 
going "memos." One available in-basket test, for example, called the 
Consolidated Fund In-Basket, has the person assume the role of a paid com- 
munity fund director. 

The colleges that opt for the use of senior comps of the sort outlined 
here, it seems to me, have resolved that their degrees are, in fact, to mean 
something (other than an accumulation of credit-hours) . Institutions in a 
multicampus system that use a reasonably standard set of senior comps have 
the further advantage of having their degrees mean roughly the same thing; 
e.g., the AB in business from Southwestern State College would mean roughly 
the same as the same degree from Southeastern State, which it seems to me is 
as it should be. In another sense, senior comps are an embodiment and 
operational definition of some of the most central of the institution's goals. 

So far we've been considering measurement for the purpose of providing 
exit criteria, for awarding degrees, for certifying. What about measurement 
for the purpose of judging institutional effectiveness? 

Intellectual Competence and College Effectiveness 

Let us understand "effectiveness," or "degree of effectiveness, " as 
the extent of achievement of institutional goals. The process of determining 



7 

While there could be different content emphases, e.g., in business 
administration curricula, the general quality-->in the sense of difficulty 
level — of the degree from one campus to another would be controlled. 
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the extent of goal achievement is generally known as "evaluation. " Let us 
further assume that some concept of intellectual development is a central 
goal at most American colleges. Note I say intellectual development , not 
intellectual status , say, at graduation. Even though a university may 
uniformly graduate highly able people, it may be that it has recruited and 
admitted only the highly able to begin with; having induced relatively 
little growth in the meantime, it would be regarded as a relatively 
ineffective institution. The college that admits students of relatively 
modest ability, and manages to increase their competency by (an average of) 

20 ability points, is rather more deserving of public support, one could 
argue, than the university which graduates the cream,, having in the process 
made them only 5 (ability) points creamier. 

To make these kinds of determinations, of course, requires repeated 
measurement- -pre- and .posttests before and after the "treatment" (in the 
language of the experimentalist --or clinician). In terms of the kinds of 
measures I've been talking about, a college would probably want to administer 
the area tests, the problem-solving exercises, and the literacy tests to 
freshmen at the time they arrive at the college. They would be given the 
specialized or departmental exams at the time they begin their major field 
work — typically at the beginning of their junior year. (These test adminis- 
trations could have diagnostic and counseling uses, in helping students 
pinpoint strengths and weaknesses and then planning their programs accord- 
ingly.) All the measures would be repeated at the end of the senior year 
(the "senior comps"). Finally, and this may seem wildly unfeasible, all 
the measures should be administered to the people five years (or so) afber 
graduation, on the assumption that there is little value in learning something if 
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it is forgotten soon after leaving college. (Differences between senior and 
"alumni" test performance could be interpreted in terms of how well the 
material had been learned originally, and also in terms of how strong a 
capacity and confidence for continued learning had been instilled during the 
four years on the campus . ) 

There could be problems in interpreting the gain (or loss) scores between 
the various administrations, and making judgments about institutional effec- 
tiveness, for many institutions in the beginning stages of the kind of 
evaluation program I've outlined. One problem is that there would be no 

Q 

available gain-score norms against which a college could interpret its gains. 
Assuming a system-wide evaluation set-up, institutions in multicampus systems 
or consortia could compare their gains against system-wide gain norms (and/or 
against one another). Otherwise, colleges could compare the gain scores of 
successive graduating classes, and they could always make comparisons between 
graduates of different departments or other such academic divisions. 

I'm sure that all this sounds prohibitive in view of the costs and student- 
hours involved. However, it may not be as bad as it seems. For a number of 
reasons, colleges, on the average, are going to get larger in the years ahead. 
With graduating classes of, say, 2,000, it should not be too difficult to 
assemble a sample of 500 alums -five-years -out at several test centers in the 
region, and then distribute (spiral) some ten or so tests among the 500--say, 
five per person. The same could be done with the freshman and senior test- 
ing--that is, sample students and test segments --providing one doesn't want 



Between the senior and alumni assessments, there are likely to be 
losses rather than gains, at least for people not going on to graduate 
school. 
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test data on all freshmen for counseling purposes, or scores for every senior 
to determine degree eligibility. 

As readers can maybe tell, I'm intrigued by this idea of colleges 

following-up a carefully selected sample of their graduates five years or 

so after graduation. In addition to taking the kinds of tests I've mentioned, 

alums could also fill out a questionnaire about career patterns, graduate 

9 

degrees, other noteworthy accomplishments, and so forth. Needless to say, 
the alums tested would be well paid for their time. 

Further down the road, should state systems decide to move along the 
lines I've outlined, they would certainly be expected to allocate sufficient 
funds to implement the kind of evaluation that commands respect on the respec- 
tive campuses. Even further on down the road, in the event the Federal 
Government decides to subsidize colleges in part on the basis of demonstrated 
effectiveness, it (Washington) would be expected to foot a sizeable part of 
the evaluation bill. 

Summary and Conclusions 

The basics of what I've suggested regarding definition and measurement 
of intellectual competency are summarized in the matrix below. 
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Colleges often do mail surveys of alumni; their value is usually limited 
however, because of low and biased return rates. 

■^Effectiveness, as I suggested earlier , should be conceived in terms of 
achievement of acknowledged (on- and off-campus) institutional goals. Nowhere 
in this paper have I meant to imply that (narrow) academic criteria need be 
the only standards for evaluating the performance of colleges and universities 
Liberal arts colleges often assert the aim of developing the "whole" man/ 
woman. Universities are generally committed to research, graduate education, 
and public service; junior colleges, to meeting a variety of local community 
neieds; technical institutes, to narrow vocational training; and so forth. 

More broadly, a government that stands for social/cultural pluralism would, 
one would expect, also stand for educational plur ali sm^ 
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INTELLECTUAL COMPETENCE 

Measured 



Academic Mastery 

1. Specialized knowledge Specifications set by college 

departments, tests constructed by 
institution-based test specialists; 
or, published achievement tests. 

2. General knowledge. Published "area" tests (of knowledge 

in the humanities, social and 
natural sciences, etc.) 

3. Literacy Published tests of reading, writing, 

and mathematical ability; or locally 
constructed tests. 



Prototype instruments developed by 
measurement organizations ( e . g . , 
ETS); additional forms, variations, 
etc. prepared by institution-based 
specialists . 

As an element in a systematic evaluation of institutional effectiveness, 
it was suggested that the various measures be taken at the beginning of the 
freshman year, at the end of the senior year, and then five years (or 
thereabouts) after graduation. The fundamental criterion of effectiveness 
would be the amount of gain ("value added," in the economist's language) in 
intellectual competency during students' undergraduate years. Additional 
criteria would be (l) the extent to which knowledge and problem solving 
skills are retained (five years) after graduation, and (2) the quality of 
other post-BA educational, occupational, cultural, etc. accomplishments. 



Intellectual Resourcefulness 



4. Skill: problem definition 

5. Capacity: new learning 

6. Creativity: problem solutions 

7. Capacity: cooperative work 






The test of specialized knowledge would be first given at the time the 
student begins his major fieldwork. 
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I have commented only on student academic and intellectual outcomes, 
which seem to me relatively noncontroversial and amenable to consensus on 



the campus and in the departments. I've said nothing at all about non- 
intellectual outcomes for students — values, commitments, self-insights — nor 
about noninstructional activities such as research and public service; all 
of these are goals that many people on many compuses would have their 
institutions embrace. While growth in the intellectual competency of 
undergraduates is undoubtedly a central goal at most colleges, it is only 
one in a larger set of priorities that institutions will need to come to 
grips with as American society, and with it, American higher education, 
evolves in the years just ahead. 



